Multimodal Dialogue Management for Multiparty Interaction with Infants

We present dialogue management routines for a system to engage in multiparty agent-infant interaction. The ultimate purpose of this research is to help infants learn a visual sign language by engaging them in naturalistic and socially contingent conversations during an early-life critical period for language development (ages 6 to 12 months) as initiated by an artificial agent. As a first step, we focus on creating and maintaining agent-infant engagement that elicits appropriate and socially contingent responses from the baby. Our system includes two agents, a physical robot and an animated virtual human. The system's multimodal perception includes an eye-tracker (measures attention) and a thermal infrared imaging camera (measures patterns of emotional arousal). A dialogue policy is presented that selects individual actions and planned multiparty sequences based on perceptual inputs about the baby's internal changing states of emotional engagement. The present version of the system was evaluated in interaction with 8 babies. All babies demonstrated spontaneous and sustained engagement with the agents for several minutes, with patterns of conversationally relevant and socially contingent behaviors. We further performed a detailed case-study analysis with annotation of all agent and baby behaviors. Results show that the baby's behaviors were generally relevant to agent conversations and contained direct evidence for socially contingent responses by the baby to specific linguistic samples produced by the avatar. This work demonstrates the potential for language learning from agents in very young babies and has especially broad implications regarding the use of artificial agents with babies who have minimal language exposure in early life

Bilingual and monolingual brains compared: A functional magnetic resonance imaging investigation of syntactic processing and a possible “Neural Signature” of bilingualism

Abstract & Does the brain of a bilingual process language differently from that of a monolingual? We compared how bilinguals and monolinguals recruit classic language brain areas in response to a language task and asked whether there is a ''neural signature'' of bilingualism. Highly proficient and early-exposed adult SpanishEnglish bilinguals and English monolinguals participated. During functional magnetic resonance imaging (fMRI), participants completed a syntactic ''sentence judgment task'' [Caplan, D., Alpert, N., & Waters, G. Effects of syntactic structure and propositional number on patterns of regional cerebral blood flow. Journal of Cognitive Neuroscience, 10, 541-552, 1998]. The sentences exploited differences between Spanish and English linguistic properties, allowing us to explore similarities and differences in behavioral and neural responses between bilinguals and monolinguals, and between a bilingual's two languages. If bilinguals' neural processing differs across their two languages, then differential behavioral and neural patterns should be observed in Spanish and English. Results show that behaviorally, in English, bilinguals and monolinguals had the same speed and accuracy, yet, as predicted from the SpanishEnglish structural differences, bilinguals had a different pattern of performance in Spanish. fMRI analyses revealed that both monolinguals (in one language) and bilinguals (in each language) showed predicted increases in activation in classic language areas (e.g., left inferior frontal cortex, LIFC), with any neural differences between the bilingual's two languages being principled and predictable based on the morphosyntactic differences between Spanish and English. However, an important difference was that bilinguals had a significantly greater increase in the blood oxygenation level-dependent signal in the LIFC (BA 45) when processing English than the English monolinguals. The results provide insight into the decades-old question about the degree of separation of bilinguals' duallanguage representation

Bilingual and monolingual brains compared: A functional magnetic resonance imaging investigation of syntactic processing and a possible “Neural Signature” of bilingualism

Abstract & Does the brain of a bilingual process language differently from that of a monolingual? We compared how bilinguals and monolinguals recruit classic language brain areas in response to a language task and asked whether there is a ''neural signature'' of bilingualism. Highly proficient and early-exposed adult SpanishEnglish bilinguals and English monolinguals participated. During functional magnetic resonance imaging (fMRI), participants completed a syntactic ''sentence judgment task'' [Caplan, D., Alpert, N., & Waters, G. Effects of syntactic structure and propositional number on patterns of regional cerebral blood flow. Journal of Cognitive Neuroscience, 10, 541-552, 1998]. The sentences exploited differences between Spanish and English linguistic properties, allowing us to explore similarities and differences in behavioral and neural responses between bilinguals and monolinguals, and between a bilingual's two languages. If bilinguals' neural processing differs across their two languages, then differential behavioral and neural patterns should be observed in Spanish and English. Results show that behaviorally, in English, bilinguals and monolinguals had the same speed and accuracy, yet, as predicted from the SpanishEnglish structural differences, bilinguals had a different pattern of performance in Spanish. fMRI analyses revealed that both monolinguals (in one language) and bilinguals (in each language) showed predicted increases in activation in classic language areas (e.g., left inferior frontal cortex, LIFC), with any neural differences between the bilingual's two languages being principled and predictable based on the morphosyntactic differences between Spanish and English. However, an important difference was that bilinguals had a significantly greater increase in the blood oxygenation level-dependent signal in the LIFC (BA 45) when processing English than the English monolinguals. The results provide insight into the decades-old question about the degree of separation of bilinguals' duallanguage representation

Age of first bilingual language exposure as a new window into bilingual reading development

Santa Maria sopra Minerva, monument to Benedetto XIII, by C. Marchinonni; Published Torino: C. Crudo, 1911-1913. Physical description: 3 v. illus. (includes plans and photographs) 319 plates. In portfolios; pt.1 Chiese; pt.2 Palazzi; pt.3 Fontane e ville. In Italian and French. Source: University of Toronto Libraries; http://main.library.utoronto.ca/ (accessed 12/22/2007

Shining new light on the brain’s “bilingual signature”: a functional near infrared spectroscopy investigation of semantic processing. Neuroimage 39:1457–1471

Decades of research have shown that, from an early age, proficient bilinguals can speak each of their two languages separately (similar to monolinguals) or rapidly switch between them (dissimilar to monolinguals). Thus we ask, do monolingual and bilingual brains process language similarly or dissimilarly, and is this affected by the language context? Using an innovative brain imaging technology, functional Near Infrared Spectroscopy (fNIRS), we investigated how adult bilinguals process semantic information, both in speech and in print, in a monolingual language context (one language at a time) or in a bilingual language context (two languages in rapid alternation). While undergoing fNIRS recording, ten early exposed, highly proficient Spanish-English bilinguals completed a Semantic Judgment task in monolingual and bilingual contexts and were compared to ten English monolingual controls. Two hypotheses were tested: the Signature Hypothesis predicts that early, highly proficient bilinguals will recruit neural tissue to process language differently from monolinguals across all language contexts. The Switching Hypothesis predicts that bilinguals will recruit neural tissue to process language similarly to monolinguals, when using one language at a time. Supporting the Signature Hypothesis, in the monolingual context, bilinguals and monolinguals showed differences in both hemispheres in the recruitment of DLPFC (BA 46/9) and IFC (BA 47/11), but similar recruitment of Broca's area (BA 44/45). In particular, in the monolingual context, bilinguals showed greater signal intensity in channels maximally overlaying DLPFC and IFC regions as compared to monolinguals. In the bilingual context, bilinguals demonstrated a more robust recruitment of right DLPFC and right IFC. These findings reveal how extensive early bilingual exposure modifies language organization in the brain-thus imparting a possible "bilingual signature." They further shed fascinating new light on how the bilingual brain may reveal the biological extent of the neural architecture underlying all human language and the language processing potential not fully recruited in the monolingual brain

“One glove does not fit all” in bilingual reading acquisition: Using the age of first bilingual language exposure to understand optimal contexts for reading success

In teaching reading, educators strive to find the balance between a code-emphasis approach and a meaning-oriented literacy approach. However, little is known about how different approaches to literacy can benefit bilingual children’s early reading acquisition. To investigate the novel hypothesis that children’s age of first bilingual exposure can interact with different approaches to literacy, we tested 56 Spanish-English bilingual children (ages 7–9), with birth exposure to Spanish and either early (before age 3) or late (3–4) age of first bilingual exposure to English. The children attended reading programs identified with either phonics or whole language emphasis. Consistent with our hypothesis, differential outcomes were linked to different ages of first bilingual exposure. Early bilingual exposure to English was associated with more advanced reading abilities under whole language emphasis, while later (ages 3–4) exposure was associated with better decoding and reading abilities under phonics emphasis. The findings show that knowing the age of a child’s first bilingual language exposure, as it corresponds to different periods in child development, may contribute to an educator’s design of reading instruction that best accommodates young bilingual learners

Exploring Cognitive Functions in Babies, Children & Adults with Near Infrared Spectroscopy

An explosion of functional Near Infrared Spectroscopy (fNIRS) studies investigating cortical activation in relation to higher cognitive processes, such as language1,2,3,4,5,6,7,8,9,10, memory11, and attention12 is underway worldwide involving adults, children and infants 3,4,13,14,15,16,17,18,19 with typical and atypical cognition20,21,22. The contemporary challenge of using fNIRS for cognitive neuroscience is to achieve systematic analyses of data such that they are universally interpretable23,24,25,26, and thus may advance important scientific questions about the functional organization and neural systems underlying human higher cognition

Fingerspelling as a Novel Gateway into Reading Fluency in Deaf Bilinguals

<div><p>Studies have shown that American Sign Language (ASL) fluency has a positive impact on deaf individuals’ English reading, but the cognitive and cross-linguistic mechanisms permitting the mapping of a visual-manual language onto a sound-based language have yet to be elucidated. Fingerspelling, which represents English orthography with 26 distinct hand configurations, is an integral part of ASL and has been suggested to provide deaf bilinguals with important cross-linguistic links between sign language and orthography. Using a hierarchical multiple regression analysis, this study examined the relationship of age of ASL exposure, ASL fluency, and fingerspelling skill on reading fluency in deaf college-age bilinguals. After controlling for ASL fluency, fingerspelling skill significantly predicted reading fluency, revealing for the first-time that fingerspelling, above and beyond ASL skills, contributes to reading fluency in deaf bilinguals. We suggest that both fingerspelling—in the visual-manual modality—and reading—in the visual-orthographic modality—are mutually facilitating because they share common underlying cognitive capacities of word decoding accuracy and automaticity of word recognition. The findings provide support for the hypothesis that the development of English reading proficiency may be facilitated through strengthening of the relationship among fingerspelling, sign language, and orthographic decoding en route to reading mastery, and may also reveal optimal approaches for reading instruction for deaf and hard of hearing children.</p></div